1. Field of the Invention
This invention relates to a vision substitution method and device for communicating audio and tactile representations of visual features found within visual representations.
2. Prior Art
Devices have previously been invented that substitute for aspects of vision with another sense, particularly hearing and touch. Often such devices convey particular pieces of information, such as the presence of obstacles. Such devices can be useful to blind and partially sighted people, and to those temporarily unable to see, for example aircraft pilots blinded by smoke. Relief images such as tactile maps can display unchanging two-dimensional images. However the instantaneous production of vision substitution images is difficult to achieve, as hearing and touch cannot fully replace the vast amount of information provided by sight. Fourier d'Albe's 1913 Reading Optophone presented the shapes of letters by scanning lines of type with five vertically-arranged spots of light, with each spot controlling the volume of a different musical note, producing characteristic sounds for each letter. The Optacon™ uses a vibrating matrix to display letters and other printed material. U.S. Pat. No. 5,097,326 describes a sound-based device for presenting live images, and U.S. Pat. No. 6,963,656 describes using musical sequences to convey features of images. The software iFeelPixel™ (described on Internet website www.ifeelpixel.com) presents audiotactile effects on force-feedback devices in response to particular graphic elements. U.S. Pat. No. 1,647 describes Morse code. There are many patents for braille displays.
Haptic joysticks and mice are disclosed in several earlier patents. U.S. Pat. No. 5,589,854 discloses a force-feedback joystick. U.S. Pat. No. 5,742,278 discloses a force effect software application programming interface and a force-feedback joystick similar to Microsoft's™ Sidewinder Force Feedback Pro™. U.S. Pat. No. 6,191,774 discloses a force-feedback mouse similar to Logitech's™ Wingman Force Feedback Mouse™. U.S. Pat. No. 6,429,849 discloses a force-feedback joystick similar to Microsoft's Force Feedback 2™.
The terms “vision substitution” or “sensory substitution” are commonly used to refer to systems that present features of images via other senses, but the terms are controversial as such systems generally only communicate a small subset of features of images. “Perceptual supplementation” has been suggested as a more appropriate term.
Since 1997 the present inventor has disclosed information on Internet websites and award entries. Internet website ourworld.compuserve.com/homepages/DavidDewhurst/homepage.htm and linked pages of Mar. 11, 1997 disclosed presenting monochrome light levels and their arrangement as tone- and speech-sounds, and their braille equivalents; simulating the function of the eye's macula and fovea, and their movement; and encoded entities. Saatchi & Saatchi “Innovation in Communication Award” entry of May 1, 1998 and follow-up details of May 8, 1998 disclosed using moving audio and tactile effects (known as “tracers”) to convey specific shapes; a haptic joystick-like device used to present shapes and used by the user to select areas of images; designer-selected features for prepared material; presenting change, and using it to control image selection; categorically-conveyed colors, textures and distances; and presenting entities as “audiographics” comprising several “tracer” components. Internet website ourworld.compuserve.com/homepages/DavidDewhurst/Vuphonics.html and linked pages of Feb. 22, 1999 disclosed audiotactile objects & structures; using intensity variation to communicate textures; and multiple movable “viewzones” of various shapes and configurations within images. (Unpublished UK Patent application Ser. No. GB9809511.1 filed May 1, 1998 covers these features.) Summary versions of the above have been accessible via the Internet websites www.vuphonics.com, www.stve.org and www.hfvc.org.
The earlier inventions did not emulate the way in which sighted people's seeing changes when they are engaged in different activities, and how their visual attention can be attuned to particular features when they are looking for certain items. For example, if blind users are seeking a large red item, they will want to be alerted to those visual features that correspond to such an item. Other parameters need to be changed when a particular activity is occurring. For example, if a widescreen-format television program is being presented, the source image will need to be masked and processed in a different way to that used for general video images.
Another problem with the earlier inventions was that the output tactile displays (e.g. braille and Optacon displays) were expensive, and it is sometimes difficult to “read” braille or other finger-read categorical tactile displays when they are embedded in a moving mouse or joystick, necessitating a separate display that is read with a separate hand. Many blind people are unable to read braille.
Earlier inventions have attempted to present shapes, but when this is done via moving effects there is a tendency for certain differing shapes to be perceived as similar “blobs”.